Unfortunately, in contrast to when I first came up with the deps-cuda solution, the sweet spot where we could easily combine CUDA dependencies for both Torch and TF seems to have vanished now, resulting in a much larger image.

It looks like an older version of nvidia-tensorflow might work better, but I'll have to do further analysis.

Update: we definitely need to hold at nvidia-tensorflow==2.15.5+nv22.11, because all later versions depend on CUDA>11.8, but 11.8 is the last version which is supported by Tensorflow 2.x on Python 3.8, so either our ocrd/core-cuda must use that, or we have to start bifurcating even that (i.e. ocrd/core-cuda11 vs ocrd/core-cuda12).

Unfortunately, the breaking changes with recent Numpy pose an additional difficulty: obviously, the older releases of TF etc are incompatible and so Numpy must now be held at <1.24 (which more recent releases of nvidia-tensorflow also ensure, but we must do post-hoc).

The whole situation will eventually get easier, with TF starting to require its CUDA dependencies explicitly, and not requiring Conda anymore, but instead allowing pip install tensorflow[and-cuda] – but again, that's unfortunately only available from TF 2.14 onwards, which is not supported for Py38.

I think I did find a workable compromise again. Let's see what the CD says.

Looking at this horde of CI jobs: shouldn't we rule out these tests if the only changes affect the dockerfiles (or docker recipes)?

I think we really need this. Even if we don't use the variant images (core-cuda-tf1, core-cuda-tf2, core-cuda-torch) yet: we need the new CUDA compromise (core-cuda) and we can reuse the new deps-tf1 rule for the TF1 venv in ocrd_all.

I resolved the conflict, but the CI failure seems unrelated – we probably just need to update test assets...

we probably just need to update test assets...

Guessed right.